{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a3f10c0",
   "metadata": {},
   "source": [
    "# Groundedness Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use the Groundedness evaluator to assess whether AI-generated responses are grounded in the provided context. The evaluator supports multiple input formats including:\n",
    "- Simple response and context evaluation\n",
    "- Query, response, and context evaluation\n",
    "- Agent responses with tool calls (file_search)\n",
    "- Multi-turn conversations\n",
    "\n",
    "This notebook uses consistent examples with the ToolCallAccuracyEvaluator for better comparison across evaluators.\n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure OpenAI Endpoint to be used for evaluation\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure OpenAI Key to be used for evaluation\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure OpenAI API version to be used for evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe3af2a1",
   "metadata": {},
   "source": [
    "## What is Groundedness?\n",
    "\n",
    "The Groundedness evaluator assesses the correspondence between claims in an AI-generated response and the source context. It ensures that responses are substantiated by the provided context, preventing hallucinations and unsupported claims.\n",
    "\n",
    "**Key Points:**\n",
    "- Even factually correct responses are considered ungrounded if they can't be verified against the provided context\n",
    "- Essential for RAG (Retrieval-Augmented Generation) applications\n",
    "- Helps ensure AI responses are trustworthy and verifiable\n",
    "\n",
    "**Scoring:** Groundedness scores range from 1 to 5, with:\n",
    "- **1**: Completely ungrounded - no claims supported by context\n",
    "- **2**: Mostly ungrounded - few claims supported\n",
    "- **3**: Partially grounded - some claims supported\n",
    "- **4**: Mostly grounded - most claims supported\n",
    "- **5**: Fully grounded - all claims supported by context"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8a00fb",
   "metadata": {},
   "source": [
    "## Groundedness Evaluator Input Requirements\n",
    "\n",
    "The Groundedness evaluator supports multiple input formats:\n",
    "\n",
    "1. **Basic Context Evaluation:**\n",
    "   - `response`: The AI response to evaluate (str)\n",
    "   - `context`: The source context/documents (str)\n",
    "   - `query`: Optional query for enhanced evaluation (str)\n",
    "\n",
    "2. **Agent Tool Evaluation:**\n",
    "   - `query`: The user query (str)\n",
    "   - `response`: Agent response with tool calls (List[dict])\n",
    "   - `tool_definitions`: Available tools, only file_search supported (List[dict])\n",
    "\n",
    "3. **Conversation Evaluation:**\n",
    "   - `conversation`: Multi-turn conversation with context (Conversation object)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa68c67",
   "metadata": {},
   "source": [
    "### Initialize Groundedness Evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "298d067a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import GroundednessEvaluator, AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "# Configure the model\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "# Initialize the evaluator\n",
    "groundedness_evaluator = GroundednessEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18b0a169",
   "metadata": {},
   "source": [
    "## Sample Evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b73b1ccd",
   "metadata": {},
   "source": [
    "### Response and Context as Strings (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e5fa898",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of a well-grounded response using weather context\n",
    "context = \"\"\"Current weather data for Seattle shows rainy conditions with a temperature of 14°C. \n",
    "The forecast indicates overcast skies with light precipitation typical for Pacific Northwest weather. \n",
    "Humidity is at 85% with winds from the southwest at 12 mph. Visibility is reduced to 8 miles due to rain.\"\"\"\n",
    "\n",
    "response = \"The current weather in Seattle is rainy with a temperature of 14°C. It's typical Pacific Northwest weather for this time of year with overcast skies and light precipitation.\"\n",
    "\n",
    "result = groundedness_evaluator(response=response, context=context)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c03f745",
   "metadata": {},
   "source": [
    "### Example of Ungrounded Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6de6a51e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of ungrounded response with unsupported claims\n",
    "context = \"\"\"Current weather data for Seattle shows rainy conditions with a temperature of 14°C. \n",
    "The forecast indicates overcast skies with light precipitation typical for Pacific Northwest weather.\"\"\"\n",
    "\n",
    "response = \"The current weather in Seattle is rainy with a temperature of 14°C. The city is experiencing its wettest month in 50 years, and the mayor has declared a weather emergency due to flooding concerns.\"\n",
    "\n",
    "result = groundedness_evaluator(response=response, context=context)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9052eb13",
   "metadata": {},
   "source": [
    "### Query, Response, and Context as Strings (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c77316c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "\n",
    "context = \"\"\"Weather report for Seattle, Washington: Currently experiencing rainy weather with temperature at 14°C. \n",
    "Overcast conditions with light rain are expected to continue. The current conditions are typical for the Pacific Northwest region \n",
    "during this season. Wind speed is moderate at 12 mph from southwest direction.\"\"\"\n",
    "\n",
    "response = \"The weather in Seattle is rainy with a temperature of 14°C. These are typical Pacific Northwest conditions with overcast skies.\"\n",
    "\n",
    "result = groundedness_evaluator(response=response, context=context, query=query)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd0a420f",
   "metadata": {},
   "source": [
    "### Query as String (str), Response as List[dict], Tool Definitions as List[dict]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72b922ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you get me the current weather information for Seattle?\"\n",
    "\n",
    "# Agent response with file_search tool call\n",
    "agent_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_filesearch_weather_123\",\n",
    "                \"name\": \"file_search\",\n",
    "                \"arguments\": {\"query\": \"current weather Seattle temperature conditions\"}\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"tool\",\n",
    "        \"tool_call_id\": \"call_filesearch_weather_123\", \n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_result\",\n",
    "                \"tool_result\": {\n",
    "                    \"content\": \"Seattle weather report: Currently rainy with temperature of 14°C. Overcast skies with light precipitation. Typical Pacific Northwest weather with 85% humidity and southwest winds at 12 mph.\"\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": \"Based on the weather data, Seattle is currently experiencing rainy weather with a temperature of 14°C. The conditions include overcast skies and light precipitation, which is typical for the Pacific Northwest.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"file_search\",\n",
    "        \"description\": \"Search through uploaded files to find relevant information\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\", \n",
    "            \"properties\": {\n",
    "                \"query\": {\"type\": \"string\", \"description\": \"Search query\"}\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "]\n",
    "\n",
    "result = groundedness_evaluator(\n",
    "    query=query, \n",
    "    response=agent_response, \n",
    "    tool_definitions=tool_definitions\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dc31ae0",
   "metadata": {},
   "source": [
    "### Conversation as Dict with Context and Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f37ae51",
   "metadata": {},
   "outputs": [],
   "source": [
    "conversation = {\n",
    "    \"context\": \"Weather data shows Seattle currently has rainy conditions at 14°C with overcast skies. London shows cloudy weather at 8°C with partly cloudy conditions. Both cities are experiencing typical seasonal weather patterns.\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Can you check the weather in Seattle for me?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"According to the current weather data, Seattle is experiencing rainy conditions with a temperature of 14°C and overcast skies.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"How does that compare to London?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"London is currently cloudier but drier than Seattle, with a temperature of 8°C and partly cloudy conditions. Seattle is warmer but rainier at 14°C.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\", \n",
    "            \"content\": \"Can you email me a summary of both cities' weather?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"I can provide you with a weather summary: Seattle has rainy weather at 14°C with overcast skies, while London has partly cloudy conditions at 8°C. However, I would need email access to send this information to you.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "result = groundedness_evaluator(conversation=conversation)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
